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Abstract 

The purpose of this study was to assist an instructor in facilitating an online course while the class was 
being offered, and to provide timely interventions for improving the course during the semester. A formative 
approach was used to help deal with unforeseen issues in implementing the first online course this instructor had 
offered. Through data analysis of what worked well, what did not work well, and what improvements were needed, 
issues related to both students and instructor's perception of the class were identified in the findings. From a 
student perspective, these issues included : online interactions, assessment, and course participation. For the 
instructor, there were concerns about new pedagogy , technology, and workload. Based on these issues, instructional 
interventions were suggested and an overall evaluation was conducted through an online survey. The approach 
adopted, findings identified, and recommendations made in this study will have implications for other instructors 
and instructional designers, especially those teaching online for the first time. 

Introduction 

The Web is offering unparalleled opportunities for student access to learning, information and 
communication, and thus has growing popularity as a primary medium in distance education (Crossman, 1997). 
Many colleges and universities are racing to move their conventional courses online (Mclsaac, 1998; Molenda & 
Sullivan, 2000). Despite the “virtual land rush” to grab space on the Internet and the excitement of the technology, 
many issues must be resolved before instruction can be delivered successfully via the Web. This study addressed 
some of the prominent issues encountered by an instructor offering a face-to-face course online for the first time. 

There are many challenges for faculty teaching online for the first time. Using new pedagogical strategies 
and working with unfamiliar technology can pose some unique difficulties. Often, the only feedback the instructor 
gets about a course is summative, which merely allows improvement the next time the course is offered. Instead of 
relying on data gathered at the end of the course, the researchers in this study took a more participative role by 
collecting data from multiple sources to inform interventions and to deal with issues as they arose. 

The purpose of the study was to assist an instructor in facilitating an online course and provide timely 
interventions for improving the course. This study investigated what worked well, what did not work well, and what 
could be done to improve or solve the problems. The results of this study describe how formative methodology was 
used to address pedagogical, technological, and communication concerns in the course. The findings are relevant to 
instructional designers and instructors involved in web-based distance education. 



Web-based instruction 

One phenomenon seen in the realm of Distance Education (DE) is the increased use of the Web as a 
primary delivery system (Crossman, 1997). When the Web was first used as a delivery system, DE was often 
considered to be traditional correspondence education. Instruction was (and in some cases still is) delivered through 
pages full of self- instructional text. Interactivity was quite limited (Hirumi & Bermudez, 1996). 

A number of case studies have provided some guidelines and heuristics on building WBI. But an extensive 
literature research of DE survey models in early 1998 revealed that there were no validated models specific to DE 
(Williams, Paprock, & Covington, 1999). Fitting distance learners’ needs still presents a challenge for distance 
educators given the differences of needs, ages, cultural backgrounds, interests, and educational levels (Willis & 
Dickinson, 1997). Distance educators and researchers are exploring more theoretical frameworks to guide the 
planning, design, development, and delivery of Web-based instruction. 
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Pedagogical issues that arise in WBI include information overload, “lost in hyperspace,” feelings of a lack 
of cohesiveness, and technical problems. Among institutional issues, the amount of time for an instructor to prepare 
and deliver the course seems to be a one of the biggest concerns (Hill, 1997). While some guidelines have been 
established on how to facilitate online learning (Eastmond, 1995; Harasim, 1993), there is a scarcity of research 
studies on implementation and how formative evaluation can help solve unforeseen problems. 

Formative Evaluation 

Formative evaluation (FE) is a process of collecting empirical data during the developmental stage for 
revision and improvement of instructional products (Guba & Lincoln, 1985; Weston, McAlpine, & Bordonaro, 

1995) . The primary goal of FE is to improve the quality of the product being developed so that the desired goals of 
the product will be met. The evaluation process involves assessment as well as judgment based on the specific 
information collected from the users (Beyer, 1995). Conventionally, FE has been conducted before implementing 
the instructional resources or curricula. The authors of this study, however, advocate using FE as an ongoing process 
in the implementation stage for distance education. 

Methodology 

Research framework 

Given the inexperience of the instructor in online teaching and the novelty of the course delivery system, 
improvements were needed throughout the course. To ensure the effectiveness of the class, timely interventions 
were crucial. Due to the nature and purpose of the study, action research was adopted as the framework. The action 
research framework was used to guide the researchers to understand the problem, develop strategies for solving the 
problems, and transform them into best practices. The research study involved three cycles of discovery, 
intervention, and evaluation (Byrant 1996). It was a constant process of observation, reflection, and action (Stringer, 

1996) . Action research methodology, when used in the context of Instructional Technology, involves an ongoing 
process of formative evaluation. This approach was used to gather functional feedback by learning from the event as 
they occurred and promoted positive changes in a timely manner. 

Research context 

This research study was carried out in an undergraduate business course being offered through a mid-size 
Midwestern university. The course was offered via Web-based courseware, Blackboard. The courseware was 
introduced on the campus in the fall of 1999 and was being used for the first time with this class. Fifteen students 
registered for this class. The majority of them were taking an online class for the first time. The professor had 
taught face-to-face for over twenty years but this was the first time he had taught the class at a distance. An 
instructional designer assisted with instructional strategies and technical concerns; however, there were a number of 
issues that needed to be addressed by using data gathered during the course. 

The instructor used Problem-Based Learning to facilitate student-centered, self-directed learning. His plan 
was to prompt students with real world questions and have groups lead class discussions for the particular topics 
being addressed. Class activities included individual discussions and small group assignments. Individual activities 
involved participating in online discussions by answering questions proposed by the instructor. Group activities 
consisted of primarily mini-cases appropriate to the chapter in the textbook for that particular week. The class was 
randomly divided into three groups with five students in each group. Students were graded by their performance on 
two online exams (with multiple choice questions and essay questions), individual online participation, and group 
project participation. 

Participants 

The participants for this study included all 15 students enrolled in the course, the instructor, and the 
instructional designer, who was responsible for instructional and technological support. 

Research questions 

The focus of this study was to formatively evaluate an undergraduate business course and help the 
instructor improve the course when it was being offered. The research questions guiding this study included: 

• What issues do instructor and students encounter in teaching and taking an online course for the first time? 

• How can these issues be resolved in a timely manner to improve a course as it is being offered? 

• What recommendations can be offered for instructors teaching an online course for the first time? 
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Data collection 

Observations 

Students’ postings were observed to determine the general trend of students’ participation. Postings 
included both individual replies to the class discussion questions posted by the instructor and also their discussions 
as a group member while collaborating with others to complete the group project. 

Survey 

Two surveys were designed to gather students’ feedback about the class. The first survey was administered 
before the middle of the semester via email. The survey included three general open-ended questions in the email 
survey: what went well, what problems they encountered, and what suggestions they had for improvement. Nine 
students out of 15 (60%) responded. 

The second survey was developed based on the responses from the first survey. The interventions 
suggested by the researchers, and the types of interaction in distance education include the following: learner- 
instructor, learner-learner, learner-content, and learner-technology (Moore, 1989; Hillman et al. 1994). This 
anonymous, Web-based survey was delivered by using the Assessment Tool in the course site. The survey was 
consisted of 19 multiple-choice questions and three open-ended questions. Twelve out of 15 students (80%) 
responded to the survey. 

Interviews 

The researchers carried out one semi-structured interview at the beginning of the project and two informal 
phone conversations with the instructor during the semester. The first interview was to obtain information about how 
the instructor would like to teach this class and what issues he was dealing with. Two informal phone interviews 
between the middle and the end of semester were to solicit instructor’s opinions on the collection of feedback from 
students and the proposed interventions based on the feedback. 

Document study 

Email messages sent back and forth between the instructor and the students provided another important 
source for understanding the issues surrounding this class. Eighty messages archived by the instructor were 
analyzed. 

Data analysis 

The data collected from the observations, surveys, interviews, and email messages were triangulated to 
determine the students’ and instructor’s perceptions of the class, issues raised throughout the semester, and 
recommendations for future improvement. Numerical data were tabulated by survey items and corresponding 
percentage in each item. The data analysis on qualitative data involved an iterative inductive process. The initial 
data analysis started with reading and rereading email messages and coding the small complete text units into 
indexed categories. Through iterative induction and constant comparison with categories and data, key issues and 
themes were generated. 

Findings 

The findings presented in this report reflect the process of identifying areas needed for improvement, 
recommendations for instructional interventions, and evaluating the course after the interventions. Observations and 
faculty interviews revealed issues that arose from delivering this first online course. The email survey data, web- 
based survey, and their email communications with the instructor revealed the students’ perception of the class and 
the problems they encountered. The findings are divided into three sections based on the process of formative 
approach. 

Early to Mid-Semester - Before Intervention 
Observation 

After nearly three weeks, there had been very few postings to the course message boards. There were only 
48 posts in the first three weeks of the class, an average of less than one post per student, per week. Several of these 
posts were related to technical issues such as where to post the answers to the individual questions and group 
questions. 

Besides the limited postings, students were answering the case problems by copying out of the book. Once 
one person had posted an answer to the problem statement, the responses from each of the team members tended to 
be very similar and did not add new information to the discussion. There was not much interaction occurring among 
the group members except when they were ready to post their answers to the class discussion board. In addition, one 
of the postings indicated that there was not much communication occurring in other forms during the early part of 
the semester. 
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After observing the first three weeks of the class, the researchers requested an interview with the instructor 
to determine his perception of the class and to investigate whether other channels of communication were being 
used. 

Faculty Interview 

The initial interview was conducted four weeks into the semester and lasted two hours. After having taught 
classes in the field of business and insurance for over twenty years in a face-to-face format, the professor found that 
teaching via the Web presented a number of challenges. Underlying many of the specific issues that the professor 
was dealing with was a lack of time to develop and work on the class because it was being taught in addition to his 
normal class load. 

New Pedagogy 

When first considering how to re-structure the class to take advantage of the Web-based environment, the 
professor decided that he needed a new pedagogical approach. He researched a number of instructional strategies 
and decided that a problem-based learning approach would be most appropriate for the content of the course. 
Implementing this new approach, however, proved to be more difficult than anticipated. 

The textbook traditionally used for the course was the primary resource for course material. The instructor 
adapted the textbook problems to the online environment and formed teams to work on solving a problem for each 
unit. Using this format, the professor hoped to stimulate conversation. By doing so, he believed that the students 
would become more engaged with the material and would have a stronger learning experience. 

At the time of the first interview, there were few indications that the students were working beyond the 
material in the textbook to solve the problems. The professor expressed some frustration with the lack of depth to 
the responses and did not feel confident that this new approach was as effective as he had hoped. He did note, 
however, that there might have been some ambiguity in the stated objectives and expectations that the students had 
been given. While he had made clear that he would base a portion of the grade on class and group participation, he 
stated that he needed to communicate his expectations better to the class. 

New Technology 

The technology used for teaching the class was also a factor in how well interactions were occurring. 
While the professor was extremely competent using many computer programs such as spreadsheets, word processors 
and email, he was unfamiliar with the development tools for the Web. He stated that he had done some pilot testing 
with the courseware in the previous semester. During the semester break, however, the university had upgraded to a 
new version, changing the capabilities and functionality of the program. Even though the courseware had better 
capabilities for supporting teams, there were some unforeseen consequences. 

Not having had the opportunity to test the new system, the professor did not anticipate the students’ 
confusion with what message to post under which board: the Group Discussion Board (for team use) or the Class 
Discussion Board. He felt that these difficulties were hindering the classes’ ability to communicate with each other 
and with him. Additionally, he found that he did not have the time to check all of the different areas where students 
might be communicating thus making it difficult to track where students might be having problems. 

Student Interactions 

The analysis of postings by students at an early part of the semester showed that there was very little use of 
the message boards, which were the intended mechanism for on-line discourse. When asking whether students were 
using any other mechanism (email, live chat, telephone) to communicate, the instructor stated that he did not have 
any indication that there were other methods being used. 

The limited interaction was contrary to the instructor’s goal that team members would work together to 
solve the problems. The instructor noted that students were reading the material because their answers on the 
discussion boards reflected their reading. They were not, however, interacting with each other in a way that 
contributed to the in-depth learning that the professor was trying to achieve. 

Email Survey 

The open-ended questions that were emailed to students near the middle of the semester gave insight into 
their perceptions of the course. While the questions were open ended, the responses were centered on a few key 
issues about the class. Given the professor’s uncertainties about how well his methods and the technology were 
working, it was surprising to find that nearly all of the students who responded to the survey had a very favorable 
impression of the class. 
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One of the major findings in this survey was that students liked the freedom and convenience of taking a 
class over the Web. For some students, it was the only way they could take such a class because they had work and 
family commitments. Many thought the technology was being used appropriately to meet their needs and 
appreciated the use of message boards over email or other communication options. Additionally, the courseware 
was viewed as easy to use and was liked by most of the students. 

While most of the respondents were satisfied with the overall structure and learning occurring in the course, 
there were a number of suggested improvements. Many of the changes revolved around the organization and 
expectations for class communication. Students wanted to have concrete guidelines for project work. Several 
mentioned that they were unsure about what was expected from their group in terms of final deliverables. There 
was also uncertainty about what criteria were being used to determine their individual participation in the class or 
within their teams. The most often mentioned concern from the students was that the professor did not seem to be 
giving enough feedback in the course Website. Students wanted more ongoing commentary from the instructor so 
that they would know whether they were meeting expectations. 

Mid-semester Intervention 

Based on the major trends identified, the researchers proposed several strategies to the instructor to 
improve the course. While the strategies were developed as a result of the problems, they were also designed to 
work within the limited time budget the professor had to make changes to the course. 

The first strategy researchers recommended was to use technology to improve interaction. Improving 
student interaction in the course is a very complex issue and is very important in a problem-based learning class with 
a team-based approach. Distance students needed a clear set of guidelines for how to communicate online. The 
recommended guidelines included two parts. One was to tell students what technology to use and when. The 
second component in the guidelines was to specify how the technology was to be used. The students who were less 
experienced in online communication did not appear to understand what they should post in the different message 
areas or how to use the tools effectively. 

The second strategy we recommended was to provide more feedback. Given the time pressure faced by the 
instructor, this was a more challenging problem. Addressing the issues related to student communication might help 
this by encouraging the behavior the professor had hoped to occur (more student-student interaction). If, for 
example, students knew that there were specified areas within the course site where they could get feedback, then 
the professor could centralize his responses and answers to one student’s questions which might help others in the 
class. Such an approach would prevent the instructor from having to go into each discussion area for each group and 
for the class to address specific issues. 

Third, we recommended providing specific guidelines for the projects. While the syllabus had clearly stated 
objectives for the outcomes in the class, there was little explicit information about how evaluation would occur. In a 
face-to-face situation, issues like class participation can be defined on an ongoing basis, depending on the lesson for 
that day. Online, however, rules for participation have to be more clearly defined. More precise project guidelines 
would help clarify students’ uncertainty about what was expected from them in terms of group work. 

Post Intervention Data 
Student Survey 

The student survey was designed, in part, based on issues found in the interview with the professor from 
students’ comments sent in via email. Twelve out of the fifteen students (80%) in the class completed the survey. 
The survey results helped confirm trends found in the qualitative data. The survey was designed to assess students’ 
learning experience with this online course, their opinions about the online test, and other issues that concerned 
them. Demographic data showed that the majority (83%) of the students had never taken an online course before 
this class. Interestingly, none of the students reported the courseware to be difficult to use (learner-technology 
interaction). The results students’ opinions on group work, students’ learning, online tests, and instructor’s feedback 
will be summarized below (interactions between learners, instructor, content). 
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Table 1: Summary of Survey Findings (The number in Q indicates the number of respondents) 



Questions 


Strongly 

Disagree 


Disagree 


No 

Opinionr. 


Agree 


Strong 

Agree 


Our group was able to work effectively as a team. 


8% (1) 


25% (3) 


8% ( 1 ) 


42% (5) 


17% (2) 


I learned more by doing this project collaboratively with 
other team members than I would have on my own. 


17% (2) 


50% (6) 


8% (1) 


1 7% (2) 


8% (1) 


This class met the expectations I had for learning the 
material. 


8% (1) 


42% (5) 


17% (2) 


17% (2) 


8% (1) 


I found online discussions to be beneficial to my learning 
experience. 


8% (1) 


33% (4) 


17% (2) 


42% (5) 


0% (0) 


Online 

Test 


The content of online test was a fair assessment 
of what I was expected to learn. 


8% (1) 


8% (1) 


42% (5) 


42% (5) 


0% (0) 


The format of the test was appropriate for an 
online class. 


8% ( 1 ) 


25% (3) 


33% (4) 


33% (4) 


0% (0) 


The instructor provided sufficient feedback. 


0% (0) 


33% (4) 


25% (3) 


42% (5) 


0% (0) 



The results on group work showed that 59% (7 out of 12) felt they were able to work effectively with their 
teams. When asked if working collaboratively with other team members helped them learn more than they would 
have working on their own, 67% (8 out of 12) disagreed or strongly disagreed, one responded with no opinion and 3 
agreed. Related to this finding was that 6 students (out of 1 1 who responded this question) did not think that their 
expectations for learning the material had been met through the class. Three of them agreed the class met their 
expectation, and 2 responded with no opinion. Regarding the online discussions, 5 students (41%) agreed that they 
were beneficial to their learning experience; 2 students (17%) reported no opinion, and 5 students (42%) did not 
think online discussions were beneficial to their learning. 

The class was somewhat divided over issues related to an online test. About 16% (2 students) of the class 
did not think the test content was a fair assessment of what they were supposed to learn, 42% had no opinion (5 
students), and 42% (5 students) thought the test was fair. As for appropriateness of test format for this online course 
(timed multiple choice questions and essay tests), 33% (4 students) responded disagree, no opinion, and agree 
respectively. 

Students’ opinions on instructor’s feedback were also divided. Five students indicted that the instructor 
provided sufficient feedback, 3 students responded no opinion, and 4 students did not think the feedback was 
sufficient. 

Students’ responses to the opened-ended questions revealed more detailed information about their 
perception of the course. These and the analysis results from students and instructors’ email messages will be 
summarized below. These qualitative data gathered from students’ survey informed the quantitative results of 
students’ perception of the class. 

Students’ Perspective on the Class 

Several students mentioned that they enjoyed the flexibility and convenience of taking an online course. 
They also liked the professor’s patience and understanding in dealing with problems they ran into. Some students 
enjoyed the group work and online discussions. Three prominent themes regarding to problems and concerns 
emerged from the qualitative data analysis. 

Technical concerns 

Although the courseware was relatively user-friendly, students constantly ran into technical problems 
throughout the semester. Several students had difficulties in taking the online exam. The restriction of the exam 
format and lack of flexibility in test sequence caused some students not to perform as well as they expected. Some 
students got a zero on particular sections due to technical restrictions. The instructor had to reset the test for these 
students. Other technical problems included lost connections during the test, transferring files, and posting and 
locating messages at the right place. 

Students reported in the survey that they did not think Blackboard was difficult to use. The email messages 
sent back and forth between students and instructor revealed that there were more technical problems than questions 
about assignments or reading. The instructor had to spend a great deal of time resolving these technical problems. 
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Assessment 

Students were very concerned about their grade, and how they were evaluated on the individual 
participation and group participation. Several students emailed the instructor asking what grade they might get and 
how they could improve their grade. In addition, students were concerned and some were frustrated by the format 
and technical difficulty of the online test. More than a couple of students mentioned that they were not good test 
takers even if they knew the material well. The timed exam made it more difficult. 

Group work 

This class used small groups to work on mini-projects. It is not surprising that there were some complaints 
about uneven participation. “Free ride” (no contribution to the group task but get the credit) was rather bothersome 
for the students who contributed more for the group project. Some students were concerned about this fairness issue 
in the assessment. 

A couple of other issues, not reflected by majority of the students, are worth mentioning. Two students 
thought that the online class lacked social interaction. They felt they would learn more through hands-on and face- 
to-face classes. In addition, insurance is a hard subject (a comment made by one student), and that not being able to 
discuss it in class made it hard to understand. 

Instructor's Perspective on the Class 

Through our informal conversations and semi -structured interview, the instructor expressed that formative 
evaluation was a very good reinforcement. He stated that it was good to have an independent source to provide 
feedback. Students seemed to like this idea and were quite open in talking about their issues and concerns. He 
thought a mid-term formative evaluation and recommendations were useful. If he had time to implement them, he 
projected these recommendations would have helped improve this class. Based on previous experience, the 
instructor had mixed feelings about his first online course. 

The instructor did not think that students accomplished what he wanted. He expected them to go beyond 
the textbook, but only one or two students did that (consistent with his comment earlier in the semester). He also felt 
that students did not learn as much as they would have in an on-campus class. With his on-campus face-to-face 
class, he was able to present extra material. With this Internet class, although databases were provided, very few 
students used them. But one of the things that instructor really enjoyed was the rapport he and some of his students 
established through emailing back and forth to address questions and concerns. He found this online rapport was 
quite rewarding. 

Discussion 

The researchers began the study by observing the visible interaction occurring on the course Web site. 
Analyzing the evaluation data identified three general major difficulties. First, very limited interaction occurred on 
the course message boards during early part of the semester. One of the reasons for this apparent lack of student 
involvement was that most of the students had never taken an online class before, and they had no experience in how 
that should communicate within the courseware. Additionally, although the instructor was aware of the lack of 
student involvement in the course, his limited time and heavy workload kept him from putting more effort into 
facilitating the students’ online communications. 

The second major problem was the amount of feedback students were getting from the instructor. The same 
factor, time shortage, also affected the amount of individual feedback the instructor was able to give to the students. 
The professor hoped that students would interact more with each other and would learn from these interactions, 
rather than relying on him for the majority of the feedback. 

The third major issue that emerged was related to standards for the class. Students indicated that they were not sure 
how they were being evaluated. This uncertainty was especially true for the participation portion of their grade. 
Clear and specific guidelines about how participation was being evaluated (for example, by number of message 
board postings, depth of the posts, etc.) would help clarify the students’ uncertainties. Likewise, providing explicit 
guidelines for group projects would have helped address students’ concerns about group work, which could lead to 
more participation from team members. 

There was a high degree of congruence between the instructor’s beliefs and student attitudes on these three 
major issues. The instructor, however, had a much more negative view of how the class was progressing than the 
students did. He felt that the students were going to be very disappointed with the overall experience they were 
getting from the class. In contrast, email to the research team and other survey data revealed that while students had 
specific issues they wanted to see improved, they were generally positive about the class. 
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This course presented a couple of challenges for the instructor. The first challenge was time management. 
Due to his busy schedule and heavy workload, he was not able to facilitate the class discussions and group activities 
as effectively as he wanted to. Much time was spent on answering students’ technical questions. If he had been free 
from dealing with these technical responsibilities, he would have been able to put more effort into the development 
of the course. Another challenge the instructor faced was finding the best fit of instructional strategy to this 
particular introductory course. There is very little literature about using PBL in Web-based instruction. Knowing 
how to implement PBL in an Internet class and how to facilitate group problem solving presented a great challenge 
for the instructor. With limited guidance from the literature, one had to explore the strategies through trial and error. 

Conclusion 

The instructor in this study was very motivated to try innovative teaching techniques and strategies. He 
spent time reading pedagogical literature and pilot testing the technology before teaching the class. Additionally, 
the class was being taught voluntarily, despite his already busy schedule. 

Even with a highly motivated faculty member, however, implementing an online class is a difficult task 
that requires new skills and strategies for success. More importantly, the instructor needs support from the 
university and his or her department to effectively work with students at a distance. The support needed most is 
time to plan, build the online courses, and ongoing technical support. 

As universities continue the mad dash to offer their courses online, reasonable timelines for design and 
development should be taken into account. If faculty have to use untested methods on constantly shifting 
courseware with their students, sound instructional practice would suggest that this is not good for the students, 
faculty, or, in the long run, the university. The reality is, however, that there will continue to be uncertainty about 
the best practices for online course delivery for the near future. Formatively evaluating their courses and making 
improvements throughout the semester can reduce some of the problems faced by faculty. 
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